Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Entrypoints][Refactor] Refactor llumnix entrypoints to be more modular #55

Merged
merged 12 commits into from
Oct 17, 2024
Merged

[Entrypoints][Refactor] Refactor llumnix entrypoints to be more modular #55

merged 12 commits into from
Oct 17, 2024

Conversation

s5u13b
Copy link
Contributor

@s5u13b s5u13b commented Oct 14, 2024 •
edited
Loading

Previously, we invasively added a lot of codes to the api server of llm engine to configure the llumnix environments in entrypoints. We refactor llumnix entrypoints codes in this pr. With this pr, we can configure the llumnix environments by adding a small amount of codes to the api server of llm engine.

@s5u13b s5u13b changed the title [WIP][Entrypoints] Refactor entrypoints [Entrypoints] Refactor llumnix entrypoints to be more modular Oct 15, 2024
@s5u13b s5u13b force-pushed the refactor-api-server branch from 75e6f1d to 5260f8e Compare October 15, 2024 03:15
@github-actions GitHub Actions
Copy link

prefill p25 p50 p75 p95 p99 mean
latency(ms) 29288.50 95136.88 182765.60 232690.35 254541.24 108054.65
decode p25 p50 p75 p95 p99 mean
latency(ms) 52.66 57.45 68.18 120.42 235.16 67.02

@github-actions GitHub Actions
Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 152.00 MB 160.00 MB 168.00 MB 176.00 MB 192.00 MB 312.00 MB 472.00 MB 536.00 MB 560.00 MB 688.00 MB 736.00 MB
rpc_speed(GB/s) 1.04 1.49 1.77 1.87 1.97 2.02 2.05 2.09 2.12 2.25 2.28 2.27 2.23 2.29 2.30 2.35 2.31 2.55 2.59 2.32 2.42 2.50 2.35 2.67 3.07 3.08 3.23 2.90 3.16
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 152.00 MB 160.00 MB 168.00 MB 176.00 MB 184.00 MB 192.00 MB 200.00 MB 216.00 MB 312.00 MB 344.00 MB 472.00 MB 560.00 MB
gloo_speed(GB/s) 0.90 1.52 1.94 2.14 2.42 2.44 2.52 2.65 2.66 3.05 2.55 2.78 2.85 3.56 2.98 2.70 2.66 2.83 2.49 2.72 2.64 2.34 2.74 2.27 2.62 2.57 2.54 2.54 2.30 2.26
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 152.00 MB 160.00 MB 168.00 MB 176.00 MB 192.00 MB 312.00 MB 320.00 MB 480.00 MB 544.00 MB 552.00 MB 568.00 MB
nccl_speed(GB/s) 0.20 0.44 0.67 0.90 1.03 1.44 1.35 1.54 1.74 1.62 1.47 2.08 2.43 2.09 1.54 2.91 2.26 3.58 2.96 3.33 4.83 3.62 3.17 4.82 4.41 6.54 4.20 4.60 4.02

KuilongCui
KuilongCui approved these changes Oct 15, 2024
View reviewed changes
llumnix/config/default.py Show resolved Hide resolved
llumnix/arg_utils.py Show resolved Hide resolved
llumnix/entrypoints/utils.py Outdated Show resolved Hide resolved
llumnix/entrypoints/vllm/api_server.py Outdated Show resolved Hide resolved
zhypku
zhypku reviewed Oct 15, 2024
View reviewed changes
llumnix/arg_utils.py Outdated Show resolved Hide resolved
llumnix/entrypoints/utils.py Show resolved Hide resolved
@github-actions GitHub Actions
Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 152.00 MB 160.00 MB 168.00 MB 176.00 MB 192.00 MB 200.00 MB 288.00 MB 312.00 MB 472.00 MB 496.00 MB 552.00 MB 560.00 MB 688.00 MB 728.00 MB
rpc_speed(GB/s) 1.04 1.53 1.77 1.91 2.02 2.09 2.13 2.13 2.16 2.29 2.26 2.28 2.31 2.22 2.38 2.41 2.33 2.41 2.54 2.47 2.34 2.41 2.49 2.55 2.60 2.67 3.09 2.82 2.77 3.10 3.10 3.09
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 152.00 MB 160.00 MB 168.00 MB 176.00 MB 184.00 MB 208.00 MB 232.00 MB 312.00 MB 320.00 MB 480.00 MB 544.00 MB 560.00 MB
gloo_speed(GB/s) 0.92 1.65 2.02 2.31 2.36 2.71 2.45 2.60 2.76 2.96 3.12 3.29 2.79 3.02 3.06 2.60 1.80 2.54 3.54 2.58 2.75 2.62 2.46 2.66 2.61 2.45 2.62 2.46 2.55 2.41
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 152.00 MB 160.00 MB 168.00 MB 176.00 MB 192.00 MB 280.00 MB 312.00 MB 320.00 MB 472.00 MB 536.00 MB 544.00 MB 696.00 MB 728.00 MB
nccl_speed(GB/s) 0.20 0.51 0.58 0.89 1.18 1.22 1.40 1.81 1.70 2.12 2.33 2.14 2.13 2.48 2.13 2.85 3.15 2.76 3.25 3.46 3.24 4.94 2.60 2.58 4.08 5.19 7.25 3.56 4.27 5.57 5.48

@github-actions GitHub Actions
Copy link

prefill p25 p50 p75 p95 p99 mean
latency(ms) 26588.33 92322.25 208279.76 227739.24 230946.08 113163.72
decode p25 p50 p75 p95 p99 mean
latency(ms) 51.06 55.65 66.76 121.43 283.44 69.72

@s5u13b s5u13b changed the title [Entrypoints] Refactor llumnix entrypoints to be more modular [Entrypoints][Refactor] Refactor llumnix entrypoints to be more modular Oct 17, 2024
@s5u13b s5u13b merged commit dc98279 into main Oct 17, 2024
14 checks passed
@s5u13b s5u13b deleted the refactor-api-server branch October 17, 2024 02:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@KuilongCui KuilongCui KuilongCui approved these changes

@zhypku zhypku zhypku approved these changes

@ZeldaHuang ZeldaHuang Awaiting requested review from ZeldaHuang

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@s5u13b @zhypku @KuilongCui